Architecture of networks with middleboxes

ABSTRACT

Some embodiments provide a system for implementing a logical network that includes a set of end machines, a first logical middlebox, and a second logical middlebox connected by a set of logical forwarding elements. The system includes a set of nodes. Each of several nodes includes (i) a virtual machine for implementing an end machine of the logical network, (ii) a managed switching element for implementing the set of logical forwarding elements of the logical network, and (iii) a middlebox element for implementing the first logical middlebox of the logical network. The system includes a physical middlebox appliance for implementing the second logical middlebox.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

This application is a divisional application of U.S. patent applicationSer. No. 14/595,199, filed Jan. 12, 2015, now published as U.S. PatentPublication 2015/0142938. U.S. patent application Ser. No. 14/595,199 isa continuation application of U.S. patent application Ser. No.13/678,498, filed Nov. 15, 2012, now issued as U.S. Pat. No. 8,966,024.U.S. patent application Ser. No. 13/678,498 claims the benefit of U.S.Provisional Application 61/560,279, entitled “Virtual MiddleboxServices”, filed Nov. 15, 2011. U.S. Applications 61/560,279, Ser. No.13/678,498, now issued as U.S. Pat. No. 8,966,024, and Ser. No.14/595,199, now published as U.S. Patent Publication 2015/0142938 areincorporated herein by reference.

BACKGROUND

Many current enterprises have large and sophisticated networkscomprising switches, hubs, routers, middleboxes (e.g., firewalls, loadbalancers, source network address translation, etc.), servers,workstations and other networked devices, which support a variety ofconnections, applications and systems. The increased sophistication ofcomputer networking, including virtual machine migration, dynamicworkloads, multi-tenancy, and customer specific quality of service andsecurity configurations require a better paradigm for network control.Networks have traditionally been managed through low-level configurationof individual network components. Network configurations often depend onthe underlying network: for example, blocking a user's access with anaccess control list (“ACL”) entry requires knowing the user's current IPaddress. More complicated tasks require more extensive networkknowledge: forcing guest users' port 80 traffic to traverse an HTTPproxy requires knowing the current network topology and the location ofeach guest. This process is of increased difficulty where the networkswitching elements are shared across multiple users.

In response, there is a growing movement towards a new network controlparadigm called Software-Defined Networking (SDN). In the SDN paradigm,a network controller, running on one or more servers in a network,controls, maintains, and implements control logic that governs theforwarding behavior of shared network switching elements on a per userbasis. Making network management decisions often requires knowledge ofthe network state. To facilitate management decision-making, the networkcontroller creates and maintains a view of the network state andprovides an application programming interface upon which managementapplications may access a view of the network state.

Some of the primary goals of maintaining large networks (including bothdatacenters and enterprise networks) are scalability, mobility, andmulti-tenancy. Many approaches taken to address one of these goalsresults in hampering at least one of the others. For instance, one caneasily provide network mobility for virtual machines within an L2domain, but L2 domains cannot scale to large sizes. Furthermore,retaining user isolation greatly complicates mobility. As such, improvedsolutions that can satisfy the scalability, mobility, and multi-tenancygoals are needed.

BRIEF SUMMARY

Some embodiments provide a system that allows a user to specify alogical network that includes one or more middleboxes (e.g., firewalls,load balancers, network address translators, intrusion detection systems(IDS), wide area network (WAN) optimizers, etc.). The system implementsthe logical network by distributing logical forwarding elements (e.g.,logical switches, logical routers, etc.) across numerous managedswitching elements operating on numerous physical machines that alsohost virtual machines of the logical network. In implementing such alogical network, the system of some embodiments implements differentmiddleboxes in different manners. For instance, the system may implementa first middlebox in a distributed manner (e.g., with the middleboximplemented across numerous managed middlebox elements that also operateon the physical machines alongside the managed switching elements) and asecond middlebox in a centralized manner (e.g., as a single appliance orvirtual machine, as a cluster). In some embodiments, the determinationas to whether to implement a particular middlebox in a distributed orcentralized matter is based on the state sharing requirements betweendifferent middlebox elements when the middlebox is distributed.

In some embodiments, the spectrum for possible implementation of logicalmiddleboxes into a physical network ranges from a fully distributedmiddlebox to a fully centralized middlebox, with different middleboxesimplemented at different points along such a spectrum. In addition, asingle type of middlebox may be implemented in both a centralized or adistributed fashion, including within the same managed logical network.For example, a user might want a first firewall for filtering alltraffic incoming from external networks and a second firewall forfiltering traffic between different subnets of the logical network. Insome cases, the best solution may be to implement the first firewall asa single appliance to which all external incoming traffic is forwarded,while implementing the second firewall in a distributed fashion acrossall of the physical machines on which virtual machines of the logicalnetwork are hosted.

At one end of the spectrum is a fully distributed middleboxarchitecture. In this case, the middlebox is implemented across numerousnodes (physical host machines). Each of the physical host machines, insome embodiments, hosts at least one virtual machine in the logicalnetwork containing the logical middlebox. In addition, a managedswitching element runs on each of the host machines, in order implementthe logical forwarding elements of the logical network. As a particularphysical host machine may host virtual machines in more than one logicalnetwork (e.g., belonging to different tenants), both the distributedmiddlebox and the managed switching element running on the host may bevirtualized in order to implement middleboxes and logical forwardingelements from different logical networks.

In some embodiments, a middlebox may be implemented in such adistributed fashion when minimal sharing of state (or none at all) isrequired between the middlebox instances. At least some types ofmiddleboxes are stateful, in that they establish states for connectionsbetween machines (e.g., between two virtual machines in the network,between a virtual machine in the network and an external machine, etc.).In some embodiments, the middlebox establishes a state for eachtransport layer connection (e.g., TCP connection, UDP connection). Inthe distributed case of some embodiments, a middlebox element operatingat a particular host machine creates states for the transportconnections passing through it, but does not need to share these stateswith the other middlebox elements operating on the other host machines.When the states only apply to the virtual machines hosted on theparticular host machine, and the middlebox does not need to perform anyanalysis using state information established for other virtual machines,then the middlebox may be distributed. Examples of such middleboxesinclude source network address translation (S-NAT), destination networkaddress translation (D-NAT), and firewalls.

In addition, some embodiments allow distribution of middleboxes thathave a minimal level of state sharing. For example, load balancers mayquery the machines across which they balance traffic to determine thecurrent level of traffic sent to each of the machines, then distributethis to the other load balancers. However, each load balancing elementcan run a load balancing algorithm on its own, and perform the queriesat regular intervals, rather than sharing state information with everyother load balancing element every time a packet is routed to one of thevirtual machines, or every time a transport (e.g., TCP, UDP, etc.)connection is established with one of the virtual machines.

On the other side of the spectrum is the fully centralized middleboximplementation. In such a centralized implementation, the managedswitching elements in the hosts send all traffic for the middlebox toprocess to the same middlebox appliance. This single middlebox may be aseparate physical machine or a separate virtual machine operating withinthe physical network on its own host machine (or in the same host as oneof the virtual machines in the network). When a managed switchingelement identifies that a packet should be sent to the middlebox, theswitching element sends the packet through the physical network to themiddlebox (e.g., via a tunnel). The middlebox processes the packet, thensends the packet (actually a new packet) to another managed switchingelement for processing (e.g., a pool node).

Some embodiments use such a centralized middlebox when a distributedmiddlebox would require packet sharing at data plane speeds. That is,for each traffic packet processed by a middlebox element, the elementwould have to update all of the other middlebox instances with the statechange resulting from the packet processing. Thus, each traffic packetpassing through a middlebox would result in an explosion of additionaltraffic in order to update all of the other middlebox instances.Examples of such middleboxes include IDSs and WAN optimizers. Forinstance, in order to properly monitor for intrusions, IDS processingneeds to know about all connections within the network. As such, if theIDS were distributed, new state updates would have to be sent for everypacket processed by a distributed IDS element.

As a third option, some embodiments use a cluster architecture for somemiddleboxes that is similar to the fully centralized architecture,except that the cluster acts as a centralized resource pool rather thana single physical machine. A middlebox cluster (e.g., a cluster of IDSboxes) may be beneficial in some embodiments when the network (ornetworks) using the middlebox is larger, and a single appliance may nothave enough resources (e.g., memory, processing power, etc.) to handlethe larger deployment. However, when the cluster is a middlebox thatrequires knowledge of all of the state information, then this stateinformation will be shared between the various machines in the cluster.In some embodiments, the middlebox cluster may be a better option than asingle appliance when the analysis does not require state updates on aper packet basis, but rather a per transport connection (or severalupdates per connection, while less often than per packet) basis. Inorder to perform the high-speed state sharing required, some embodimentslink the middlebox machines in the cluster via a separate dedicatedhigh-speed connection for sharing the state information.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawing, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 conceptually illustrates a spectrum for the implementation oflogical middleboxes into a physical network, ranging from a fullydistributed middlebox to a fully centralized middlebox.

FIG. 2 conceptually illustrates a logical network topology of someembodiments.

FIG. 3 conceptually illustrates a distributed middlebox implementationof some embodiments.

FIG. 4 conceptually illustrates a fully centralized implementation of amiddlebox of some embodiments.

FIG. 5 conceptually illustrates a an implementation of a middlebox as acluster of resources of some embodiments.

FIG. 6 conceptually illustrates a network that implements a logicalnetwork that includes an intrusion detection system of some embodiments.

FIG. 7 conceptually illustrates an architectural diagram of a hostmachine of some embodiments that includes both a distributed softwaremiddlebox element and a software switching element.

FIG. 8 illustrates a network control system of some embodiments forconfiguring managed switching elements and distributed middleboxelements in order to implement logical networks.

FIG. 9 conceptually illustrates the propagation of data through thenetwork control system of some embodiments.

FIG. 10 illustrates example architecture of a network controller of someembodiments.

FIG. 11 conceptually illustrates a complex logical network topologyinvolving numerous middleboxes.

FIG. 12 conceptually illustrates one particular physical implementationof the network of FIG. 11 in a hosted, virtualized environment.

FIG. 13 conceptually illustrates an electronic system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments provide a system that allows a user to specify alogical network that includes one or more middleboxes (e.g., firewalls,load balancers, network address translators, intrusion detection systems(IDS), wide area network (WAN) optimizers, etc.). The system implementsthe logical network by distributing logical forwarding elements (e.g.,logical switches, logical routers, etc.) across numerous managedswitching elements operating on numerous physical machines that alsohost virtual machines of the logical network.

In implementing such a logical network, the system of some embodimentsimplement different middleboxes in different manners. For instance, thesystem may implement a first middlebox in a distributed manner (e.g.,with the middlebox implemented across numerous managed middleboxelements that also operate on the physical machines alongside themanaged switching elements) and a second middlebox in a centralizedmanner (e.g., as a single appliance or virtual machine, as a cluster).In some embodiments, the determination as to whether to implement aparticular middlebox in a distributed or centralized matter is based onthe state sharing requirements between different middlebox elements whenthe middlebox is distributed.

FIG. 1 conceptually illustrates a spectrum 100 for the implementation oflogical middleboxes into a physical network, ranging from a fullydistributed middlebox to a fully centralized middlebox. As mentioned,different middleboxes may be implemented at different points along thisspectrum. In addition, a single type of middlebox may be implemented inboth a centralized or a distributed fashion, including within the samemanaged logical network. For example, a user might want a first firewallfor filtering all traffic incoming from external networks and a secondfirewall for filtering traffic between different subnets of the logicalnetwork. In some cases, the best solution may be to implement the firstfirewall as a single appliance to which all external incoming traffic isforwarded, while implementing the second firewall in a distributedfashion across all of the physical machines on which virtual machines ofthe logical network are hosted.

At the left end of spectrum 100 is a fully distributed middleboxarchitecture 105 of some embodiments. As shown in the diagram, themiddlebox is implemented across numerous nodes (physical host machines).Each of the physical host machines, in some embodiments, hosts at leastone virtual machine in the logical network containing the logicalmiddlebox. In addition, a managed switching element runs on each of thehost machines, in order implement the logical forwarding elements (e.g.,logical routers, logical switches) of the logical network. As aparticular physical host machine may host virtual machines in more thanone logical network (e.g., belonging to different tenants), both thedistributed middlebox and the managed switching element running on thehost may be virtualized in order to implement middleboxes and logicalforwarding elements from different logical networks. In someembodiments, the middlebox is implemented as a module or set of modulesthat runs within the hypervisor of the host machine.

As indicated in the figure, a middlebox may be implemented in such adistributed fashion when minimal sharing of state (or none at all) isrequired between the middlebox instances. At least some types ofmiddleboxes are stateful, in that they establish states for connectionsbetween machines (e.g., between two virtual machines in the network,between a virtual machine in the network and an external machine, etc.).In some embodiments, the middlebox establishes a state for eachtransport layer connection (e.g., TCP connection, UDP connection). Inthe distributed case of some embodiments, a middlebox element operatingat a particular host machine creates states for the transportconnections passing through it, but does not need to share these stateswith the other middlebox elements operating on the other host machines.When the states only apply to the virtual machines hosted on theparticular host machine, and the middlebox does not need to perform anyanalysis using state information established for other virtual machines,then the middlebox may be distributed. Examples of such middleboxesinclude source network address translation (S-NAT), destination networkaddress translation (D-NAT), and firewalls.

In addition, some embodiments allow distribution of middleboxes thathave a minimal level of state sharing. For example, load balancers mayquery the machines across which they balance traffic to determine thecurrent level of traffic sent to each of the machines, then distributethis to the other load balancers. However, each load balancing elementcan run a load balancing algorithm on its own, and perform the queriesat regular intervals, rather than sharing state information with everyother load balancing element every time a packet is routed to one of thevirtual machines, or every time a transport (e.g., TCP, UDP, etc.)connection is established with one of the virtual machines.

On the other side of the spectrum 100 is the fully centralized middleboximplementation 110. In this implementation, the managed switchingelements in the hosts send all traffic for the middlebox to process tothe same middlebox appliance. This single middlebox may be a separatephysical machine or a separate virtual machine operating within thephysical network on its own host machine (or in the same host as one ofthe virtual machines in the network). When a managed switching elementidentifies that a packet should be sent to the middlebox, the switchingelement sends the packet through the physical network to the middlebox(e.g., via a tunnel). The middlebox processes the packet, then sends thepacket (actually a new packet) to another managed switching element forprocessing.

In some embodiments, the managed switching element to which the newpacket is sent is a pool node. A pool node, in some embodiments, is aparticular type of managed switching element located at the networkinterior (i.e., not directly connected to any of the virtual machines atthe network edges) for handling traffic that the edge switching elements(i.e., those located on the host machines, directly connected to thevirtual machines) are not able to process. In other embodiments, ratherthan send the packet to a pool node, the middlebox sends the traffic toa managed switching element built directly into the middlebox appliance.

Some embodiments use such a centralized middlebox when a distributedmiddlebox would require packet sharing at data plane speeds. That is,for each traffic packet processed by a middlebox element, the elementwould have to update all of the other middlebox instances with the statechange resulting from the packet processing. Thus, each traffic packetpassing through a middlebox would result in an explosion of additionaltraffic in order to update all of the other middlebox instances.Examples of such middleboxes include IDSs and WAN optimizers. Forinstance, in order to properly monitor for intrusions, IDS processingneeds to know about all connections within the network. As such, if theIDS were distributed, new state updates would have to be sent for everypacket processed by a distributed IDS element.

The middlebox cluster architecture 115 is similar to the fullycentralized architecture 110, except that the cluster acts as acentralized resource pool rather than a single physical machine. Asshown in the figure, a middlebox cluster (e.g., a cluster of IDS boxes)may be beneficial when the network (or networks) using the middlebox islarger, and a single appliance may not have enough resources (e.g.,memory, processing power, etc.) to handle the larger deployment.However, when the cluster is a middlebox that requires knowledge of allof the state information, then this state information will be sharedbetween the various machines in the cluster. In some embodiments, themiddlebox cluster may be a better option than a single appliance whenthe analysis does not require state updates on a per packet basis, butrather a per transport connection (or several updates per connection,while less often than per packet) basis. In order to perform thehigh-speed state sharing required, some embodiments link the middleboxmachines in the cluster via a separate dedicated high-speed connectionfor sharing the state information.

The above illustrates examples of different implementation of logicalmiddleboxes in a network of some embodiments. Several more detailedembodiments are described below. Section I describes the differentmiddlebox architectures of some embodiments. Section II describes thedistributed middlebox implementation of some embodiments. Next, SectionIII describes the network control system of some embodiments forconfiguring a network in order to implement a logical network thatincludes one or more middleboxes. Section IV then describes anillustrative example of a network with numerous different middleboxes.Finally, Section V describes an electronic system with which someembodiments of the invention are implemented.

I. Different Middlebox Architectures

As described above, some embodiments implement different middleboxesusing different architectures within a managed network. Even for thesame logical network topology, some middleboxes are implemented in adistributed fashion (with a middlebox element operating on each host onwhich a virtual machine of the network operates), while othermiddleboxes are implemented in a centralized fashion (with a singleappliance or cluster to which the managed switching elements on thehosts connect).

FIG. 2 conceptually illustrates a logical network topology 200 of someembodiments. The network topology 200 is a simplified network forpurposes of explanation. The network includes two logical L2 switches205 and 210 connected by a logical L3 router 215. The logical switch 205connects virtual machines 220 and 225, while the logical switch 210connects virtual machines 230 and 235. The logical router 215 alsoconnects to an external network 250.

In addition, a middlebox 240 attaches to the logical router 215. One ofordinary skill in the art will recognize that the network topology 200represents just one particular logical network topology into which amiddlebox may be incorporated. In various embodiments, the middlebox maybe located directly between two other components (e.g.,), directlybetween the gateway and logical router (e.g., in order to monitor andprocess all traffic entering or exiting the logical network), or inother locations in a more complex network.

In the architecture shown in FIG. 2, the middlebox 240 is not locatedwithin the direct traffic flow, either from one domain to the other, orbetween the external world and the domain. Accordingly, packets will notbe sent to the middlebox unless routing policies are specified (e.g., bya user such as a network administrator) for the logical router 215 thatdetermine which packets should be sent to the middlebox for processing.Some embodiments enable the use of policy routing rules that forwardpackets based on data beyond the destination address (e.g., destinationIP or MAC address). For example, a user might specify (e.g., through anetwork controller application programming interface (API) that allpackets with a source IP address in the logical subnet switched bylogical switch 205, or all packets that enter the network from theexternal network 250 destined for the logical subnet switched by thelogical switch 210, should be directed to the middlebox 240 forprocessing.

Different middleboxes may perform different functionalities within thenetwork. For instance, a firewall analyzes data packets to determinewhether or not the packets should be allowed through (i.e., similar toACL flow entries). The firewall stores a set of rules (e.g., entered bya user) that determine whether or not the firewall drops (i.e.,discards) or allows the packet through (or, in some cases, rejects thepacket by dropping the packet and sending an error response back to thesender). In some embodiments, the firewall is a stateful firewall thatkeeps track of transport (e.g., TCP and/or UDP) connections, and usesthe stored state information to make faster packet processing decisions.

Source network address translation (S-NAT) modifies the source IPaddress of packets in the packet headers. For instance, S-NAT may beused so that the IP addresses of numerous different machines withdifferent IP addresses can be hidden from destination machines bychanging the source of packets from the different machines to a singleIP address. Destination network address translation (D-NAT) similarlymodifies the destination IP address of packets in order to hide the realIP addresses from the source machines. Load balancing is a form of D-NATthat uses various algorithms (e.g., round robin, random assignment,etc.) to balance traffic across numerous destination machines. A loadbalancer receives a packet for a specific IP address that is exposed tothe source machine and modifies the destination IP address of the packetto match up with a particular one of the destination machines selectedby the load balancing algorithm.

An intrusion detection system (IDS) is a passive middlebox in someembodiments that monitors the logical network for malicious activitiesor policy violations. The IDS may examine transport connections (e.g.,TCP connections, UDP connections, etc.) to determine whether an attackon the network is occurring.

A WAN optimizer is a middlebox device for increasing the efficiency ofdata transfers across a WAN (e.g., accelerating the flow of data acrossthe WAN). Examples of WAN optimization techniques include datadeduplication, data compression, latency optimization, caching and/orproxying, forward error correction, protocol spoofing, traffic shaping,equalizing, connection limiting, simple rate limiting, etc. While theabove is a list of some of the several different middleboxes, one ofordinary skill in the art will recognize that some embodiments mayinclude various different middleboxes that may be implemented in eithera distributed or centralized manner.

Depending on the type of middlebox and, in some cases, the type ofimplementation requested by a user, a middlebox such as that shown inFIG. 2 will be implemented in either a centralized fashion or adistributed fashion. FIG. 3 conceptually illustrates such a distributedimplementation 300 of some embodiments. Specifically, FIG. 3 illustratesseveral nodes, including a first host machine 305, a second host machine310, a third host machine 315, and an Nth host machine 320. Each of thefirst three nodes hosts several virtual machines of the network 200,with virtual machine 220 hosted on the first host machine 305, virtualmachines 225 and 235 hosted on the second host machine 310, and virtualmachine 230 hosted on the third host machine 315.

In addition, each of the host machines includes a managed switchingelement (“MSE”). The managed switching elements of some embodiments aresoftware forwarding elements that implement logical forwarding elementsfor one or more logical networks. For instance, the MSEs in the hosts305-320 include flow entries in forwarding tables that implement thelogical forwarding elements of network 200. Specifically, the MSEs onthe host machines implement logical switches 205 and 210, as well as thelogical router 215. On the other hand, some embodiments only implementlogical switches at a particular node when at least one virtual machineconnected to the logical switch is located at the node (i.e., onlyimplementing logical switch 205 and logical router 215 in the MSE athost 305. The Nth host 320 does not include any virtual machines fromthe network 200, and therefore the MSE residing on that host does notimplement any logical forwarding elements from the network 200.

The implementation 300 of some embodiments also includes a pool node 340that connects to the host machines. In some embodiments, the MSEsresiding on the host perform first-hop processing. That is, these MSEsare the first forwarding elements a packet reaches after being sent froma virtual machine, and attempt to perform all of the logical switchingand routing at this first hop. However, in some cases a particular MSEmay not store flow entries containing all of the logical forwardinginformation for a network, and therefore may not know what to do with aparticular packet. In some such embodiments, the MSE sends the packet toa pool node 340 for further processing. These pool nodes are interiormanaged switching elements which, in some embodiments, store flowentries that encompass a larger portion of the logical network than theedge software switching elements.

Similar to the distribution of the logical switching elements across thehosts on which the virtual machines of network 200 reside, the middlebox240 is distributed across middlebox elements on these hosts 305-315. Insome embodiments, a middlebox module (or set of modules) resides on thehost machines (e.g., operating in the hypervisor of the host). When theuser sets up the logical network (e.g., network 200), the input includesa configuration from the middlebox. For instance, for a firewall, theuser would input a set of rules for packet filtering (e.g., based on IPaddress, TCP connection, etc.). In some embodiments, a network controlsystem that is used to provision the managed switching elements toimplement the logical forwarding elements may also be used to provisionthe various middlebox elements operating on the host machines. When theuser inputs the middlebox configuration into a controller of the networkcontrol system, the controller identifies the particular nodes overwhich the middlebox configuration should be implemented, and distributesthe configuration to these nodes (e.g., through a set of controllers).

When one of the virtual machines sends a packet (e.g., to another one ofthe virtual machines, to an external address, etc.), the packetinitially goes to the local managed switching element for processing.The MSE may use its stored flow entries stored to make a forwardingdecision to send the packet to the middlebox, in which case someembodiments send the packet to the local middlebox element on the samehost. In some embodiments, the middlebox element and the MSE negotiate asoftware port through which to transfer packets with minimal delay.After the middlebox processes the packet, some embodiments then send thepacket back to the MSE through this same port. In some embodiments, thispacket is sent from the middlebox to the MSE as a new packet, andtherefore requiring new processing by the MSE. In some situations,however, no packets are sent back. For instance, if the middlebox is afirewall, the middlebox may block or drop the packet. In addition, someembodiments of the middlebox are passive, and duplicates of the packetsare sent to the middlebox in order for the middlebox to keep track ofstatistics, but are not sent back to the switching element.

While FIG. 3 illustrates only one logical network implemented across thehosts 305-320, some embodiments implement numerous logical networks(e.g., for different tenants) across the set of hosts. As such, amiddlebox element on a particular host might actually storeconfigurations for several different middleboxes belonging to severaldifferent logical networks. For example, a firewall element may bevirtualized to implement two (or more) different firewalls. These willeffectively operate as two separate middlebox instances, such that themiddlebox element is sliced into several “virtual” middleboxes (of thesame type). In addition, when the MSE on the host sends packets to themiddlebox, some embodiments append (e.g., prepend) a slice identifier(or tag) on the packet to identify to which of the several virtualmiddleboxes the packet is being sent. When multiple middleboxes areimplemented on the same middlebox element for a single logical network(e.g., two different load balancers), the slice identifier will need toidentify the particular middlebox slice rather than just the logicalnetwork to which the packet belongs. Different embodiments may usedifferent slice identifiers for the middleboxes.

Examples of middleboxes that may be distributed in some embodimentsinclude firewalls, S-NATs, and load balancers. In each of these cases,the middlebox plays an active role in the packet processing (i.e.,S-NATs and load balancers modify source and destination addresses of thepackets, respectively, while firewalls make decisions as to whether toallow or drop packets). However, each of these middlebox elements on aparticular node can function on its own without requiring informationfrom the corresponding middlebox elements on the other nodes. Evendistributed load balancer elements can each separately load balanceincoming traffic across different virtual machines with the assumptionthat none of the virtual machines are likely to become overloaded solong as the other load balancer elements use the same algorithm.Nevertheless, in some embodiments, the load balancer elements will sharestate (e.g., after querying the destination virtual machines for usageand health statistics) at some level.

FIG. 3, as stated, conceptually illustrates a distributed implementationof some embodiments for the middlebox 240 of logical network 200. FIG.4, on the other hand, conceptually illustrates a fully centralizedimplementation 400 of the middlebox 240. This implementation, like thedistributed example of FIG. 3, also includes several nodes 405-420 thathost the virtual machines 220-235. These virtual machines are againarranged with the first virtual machine 220 hosted on node 405, thesecond and fourth virtual machines 225 and 235 hosted on node 410, andthe third virtual machine 230 hosted on node 415. Similarly, the managedswitching elements situated on the nodes 405-415 implement the logicalforwarding elements 205-215. As in the distributed middlebox example,the managed switching elements perform first hop processing on packetsoriginating from the virtual machines.

In this example, however, the middlebox 240 is not distributed acrossthe hosts 405-415. Instead, the middlebox is implemented as a singlemachine external to the hosts. In different embodiments, and fordifferent types of middleboxes, this single box may be a single physicalappliance (e.g., a separate physical device) or a single virtual machine(which may, in fact, operate on one of the host machines or on adifferent host machine). For instance, some embodiments may provide afirst type of middlebox (e.g., a WAN optimizer) as a virtual machinewhile providing a second type of middlebox (e.g., an IDS) as a singleappliance. In addition, some embodiments may provide both options for asingle type of middlebox.

As with the distributed middlebox, in some embodiments the networkcontrol system is used to provision a centralized middlebox appliance orvirtual machine. Rather than the controller receiving configurationinformation and identifying numerous nodes to which the configurationinformation should be distributed, the controller of some embodimentsidentifies an appliance on which to implement the middlebox anddistributes the configuration to the appliance (e.g., through anintermediary controller that manages the appliance. In some embodiments,several physical appliances may exist within the physical network andthe controller chooses one of these appliances to implement themiddlebox. When the middlebox is implemented as a virtual machine, someembodiments select a host node for the virtual machine and thendistribute the configuration to the node. In either case, the networkcontrol system also specifies a connection, or attachment, between themiddlebox and the various managed switching elements on the host. Insome embodiments, the middlebox appliance supports one or more types oftunneling, and the flow entries distributed to the managed switchingelements include entries specifying the tunnel encapsulation to use inorder to send packets to the middlebox.

When a flow entry in a managed switching element specifies to sendtraffic to the middlebox, the managed switching element alsoencapsulates the packet using this tunnel information and sends thepacket out of its host through the tunnel to the middlebox. As with thedistributed middleboxes, some centralized middleboxes are activemiddleboxes. That is, the middlebox sends the packet back to the networkafter performing its middlebox processing. In some embodiments, suchmiddleboxes are configured to always send the packet (as a new packet)to a pool node (e.g., always the same pool node, one of several poolnodes). In FIG. 4, the centralized middlebox 425 sends all of itsoutgoing traffic to the pool node 430. The pool node, which alsoimplements the logical forwarding elements 205-215, then forwards thepacket to the appropriate destination machine.

Just as the distributed middlebox elements may be virtualized to performmiddlebox instances for several different logical networks, so may thecentralized middlebox 425. The same physical appliance (or virtualmachine) may be used by numerous different logical networks. In someembodiments, a similar slicing technique to that used in the distributedarchitecture is used. That is, the managed switching element adds a tagto indicate the logical network (or particular logical middlebox in thelogical network) to which the packet is being sent, and the middlebox425 uses this tag to identify which of the middlebox instances itimplements should be used to process the packet. In some embodiments,the centralized middlebox appliance includes numerous ports, each ofwhich maps to a different virtual middlebox instances. In suchembodiments, the slicing technique may not be used, and instead theincoming port is used to identify the correct virtual middlebox.

Whereas the middlebox 425 is a single resource, some embodimentsimplement a middlebox as a centralized cluster of resources, as shown inthe implementation 500 of FIG. 5. This example is the same as that shownin FIG. 4 except that, rather than a single middlebox device, thenetwork includes a middlebox cluster 505 with three middlebox resources510-520. In some embodiments, each of the middlebox resources 510-520 isa separate device or virtual machine (i.e., the equivalent of middlebox425).

Different embodiments may use different architectures within themiddlebox cluster. Some embodiments include an entry point (e.g., asingle physical device) to the cluster that load balances packets acrossthe resource pool. Other embodiments have different host machinesconnected directly to different resources within the cluster. Forinstance, the network 500 might be set up with the first host machine405 connected to resource 510, the second host machine 410 connected toresource 515, and the third host machine 415 connected to resource 520.Other embodiments use a master-backup setup in which the cluster has twodevices. The host machines all connect to the master, which performs themiddlebox processing, while sharing state data with the backup resource.

As described above by reference to FIG. 2, some embodiments use thecentralized middlebox implementation when state sharing is required on aper-packet basis. That is, for certain middleboxes, the middleboxprocessing requires knowledge of all of the packets processed by themiddlebox. For a distributed middlebox, this would require an explosionof state updates being sent out over the network connecting themiddlebox elements. However, when the middlebox is a single appliance,that single appliance will store all of the state at all times.

For a middlebox cluster such as the cluster 505, this requirement meansthat state must be shared between the middlebox resources in the clusterat high speeds. Some embodiments use dedicated connections between themiddlebox resources to share this state information. That is, aparticular port on each middlebox device is dedicated only to statesharing between the several devices in the cluster. Often, a middleboxcluster will only be two machines, or a few machines, operating in closeproximity making such dedicated connections more feasible. Forsituations with more than two middleboxes, some embodiments use a meshnetwork in which each middlebox resource broadcasts state updates overthe network to all other middlebox resources. Other embodiments use astar network in which the middlebox resources transmit their stateupdates to a central resource that amalgamates the updates and sendsthem to the other resources. While middlebox clusters require thisadditional infrastructure as compared to the centralized, case, theclusters have the benefit of being able to handle larger deployments inwhich a larger number of packets are processed.

As mentioned, both WAN optimizers and intrusion detection systems areexamples of middleboxes that some embodiments implement as centralizedmiddleboxes because of the state sharing requirements. The WANoptimizer, for example, increases the efficiency of data transfersacross a WAN using various optimization techniques. To perform theseoptimization techniques requires access to all of the traffic being sentover the WAN, and thus a centralized implementation is more optimal.Furthermore, the WAN optimizer may be used to cache content sent overthe WAN, and the caching only serves its purpose if the cache is storedtogether rather than distributed over numerous hosts.

An intrusion detection system is a passive system (i.e., does not dropor modify packets) that monitors total numbers of connections, theaddresses on those connections, the number of packets for eachconnection, etc. In order to detect intrusions, the IDS looks forpatterns in the connections, heuristics, etc., for which the IDSprocessing must be aware of all of the traffic monitored. If onedistributed element has information about a first connection and asecond distributed element has information about a second connection,neither element has enough information to properly evaluate the networkfor an intrusion.

FIG. 6 conceptually illustrates a network 600 that implements a logicalnetwork that includes an intrusion detection system 625 of someembodiments. The logical network includes four virtual machines hostedamong three nodes 605, 610, and 620. The logical topology of the logicalnetwork in this case is the same as that shown in FIG. 2 (with the IDSas the middlebox), though the aspects of the IDS implementationdescribed here apply to other network topologies as well. The network600 also includes a fourth node 615 that does not host any of thevirtual machines of the particular logical network (but does host atleast one virtual machine from another network).

Unlike the previous figures, FIG. 6 shows connections along with arrowsto show direction of packet transfer between machines in the system.Thus, for example, all of the hosts 605-620 can send packets to eachother in both directions. That is, virtual machines at host 605 sendpackets to virtual machines at 620 (via the managed switching element athost 605 and then the managed switching element at host 620), and thevirtual machines at host 620 also send packets to the virtual machinesat host 605. In addition, all of the MSEs at the hosts may use the poolnode to process packets for which the edge MSEs cannot make a forwardingdecision.

Each of the host machines 605, 610, and 620, that host virtual machinesfrom the particular logical network, send packets to the IDS. In someembodiments, all traffic on the logical network is sent to the IDS.However, these arrows are unidirectional, as the intrusion detectionsystem of some embodiments is a passive middlebox. Some embodiments,rather than forwarding traffic through the middlebox, send a duplicatepacket to the IDS box 625. The IDS receives these duplicate packets(i.e., a packet for each one sent through the network between hostsand/or an external network) and performs its intrusion detectionanalysis. Because the intrusion detection system 625 does not output anytraffic packets, there is no connection needed between the IDS 625 andthe pool node 630 in this figure.

II. Distributed Middlebox Implementation

As described above, some embodiments implement one or more differentmiddleboxes in a distributed fashion, with middlebox elements operatingin some or all of the host machines on which the virtual machines andmanaged switching elements of a logical network are located, as comparedto the centralized middlebox implementation of some embodiments. Thissection describes the distributed middlebox implementation of someembodiments within a host machine.

FIG. 7 conceptually illustrates an architectural diagram of a hostmachine 700 of some embodiments that includes both a distributedsoftware middlebox element and a software switching element. Thedistributed software middlebox element may be a network addresstranslation element, a firewall element, a load balancing element, orany other middlebox implemented in a distributed fashion.

In this example, the middlebox element includes three components on thehost machine—a middlebox daemon 790 that runs in the user space of thehost machine 700 and a middlebox kernel module 795 that runs in thekernel of the host machine 700. While this figure illustrates thedistributed middlebox element as two components for the purpose ofexplanation, the middlebox daemon 790 and the middlebox kernel module795 collectively form the middlebox element running on the host machine700. The software switching element (an open virtual switch (“OVS”), inthis example) includes three components—an OVS kernel module 745 thatruns in the kernel of the host machine 700, and an OVS daemon 765 and anOVS database (DB) daemon 767, which both run in the user space of thehost machine.

As illustrated in FIG. 7, the host 700 includes hardware 705, kernel720, user space 721, and VMs 785-795. The hardware 705 may includetypical computer hardware, such as processing units, volatile memory(e.g., random access memory (RAM)), non-volatile memory (e.g., hard diskdrives, flash memory, optical discs, etc.), network adapters, videoadapters, or any other type of computer hardware. As shown, the hardware705 includes NICs 710 and 715, which in some embodiments are typicalnetwork interface controllers for connecting a computing device to anetwork.

As shown in FIG. 7, the host machine 700 includes a kernel 720 and auser space 721. In some embodiments, the kernel is the most basiccomponent of an operating system that runs on a separate memory spaceand is responsible for managing system resources (e.g., communicationbetween hardware and software resources). In contrast, the user space isa memory space where all user mode applications may run.

The kernel 720 of some embodiments is a software abstraction layer thatruns on top of the hardware 705 and runs below any operating system. Insome embodiments, the kernel 720 performs virtualization functionalities(e.g., to virtualize the hardware 705 for several virtual machinesoperating on the host machine). The kernel 720 is then part of ahypervisor, in some embodiments. The kernel 720 handles variousmanagement tasks, such as memory management, processor scheduling, orany other operations for controlling the execution of the VMs 735 and738 operating on the host machine.

As shown, the kernel 720 includes device drivers 725 and 730 for theNICs 710 and 715, respectively. The device drivers 725 and 730 allow anoperating system (e.g., of a virtual machine) to interact with thehardware of the host 700. In this example, the device driver 725 allowsinteraction with the NIC 710, while the driver 730 allows interactionwith the NIC 715. The kernel 720 may include other device drivers (notshown) for allowing the virtual machines to interact with other hardware(not shown) in the host 700.

The virtual machines 735 and 738 are independent virtual machinesrunning on the host machine 700 (e.g., user virtual machines such asthose shown in FIGS. 3-6), using resources virtualized by the kernel720. As such, the VMs run any number of different operating systems.Examples of such operations systems include Solaris, FreeBSD, or anyother type of Unix-based operating system. Other examples includeWindows-based operating systems as well.

As shown, the user space 721 of the host machine 700 includes themiddlebox daemon 790, the OVS daemon 765, and the OVS DB daemon 767.Other applications (not shown) may be included in the user space 721 aswell, including daemons for additional distributed middleboxes (e.g.,firewalls, load balancers, network address translators, etc.). The OVSdaemon 765 is an application that runs in the user space 721. Someembodiments of the OVS daemon 765 communicate with a network controller780 in order to receive instructions, as described in further detailbelow, for processing and forwarding packets sent to and from thevirtual machines 735 and 738. The OVS daemon 765 of some embodimentscommunicates with the network controller 780 through the OpenFlowprotocol, while other embodiments use different communication protocolsfor transferring the physical control plane data. Additionally, in someembodiments the OVS daemon 765 retrieves configuration information fromthe OVS DB daemon 767 after the network controller 780 transmits theconfiguration information to the OVS DB daemon.

In some embodiments, the OVS DB daemon 767 also runs in the user space721. The OVS DB daemon 767 of some embodiments communicates with thenetwork controller 780 in order to configure the OVS switching element(e.g., the OVS daemon 765 and/or the OVS kernel module 745). Forinstance, the OVS DB daemon 767 receives configuration information fromthe network controller 780 and stores the configuration information in aset of databases. In some embodiments, the OVS DB daemon 767communicates with the network controller 780 through a databasecommunication protocol. In some cases, the OVS DB daemon 767 may receiverequests for configuration information from the OVS daemon 765. The OVSDB daemon 767, in these cases, retrieves the requested configurationinformation (e.g., from a set of databases) and sends the configurationinformation to the OVS daemon 765.

The OVS daemon 765 includes an OpenFlow protocol module 770 and a flowprocessor 775. The OpenFlow protocol module 770 communicates with thenetwork controller 780 to receive configuration information (e.g., flowentries) from the network controller 780 for configuring the softwareswitching element. When the module 770 receives configurationinformation from the network controller 780, it translates theconfiguration information into information understandable by the flowprocessor 775.

The flow processor 775 manages the rules for processing and routingpackets. For instance, the flow processor 775 stores rules (e.g., in astorage medium, such as a disk drive) received from the OpenFlowprotocol module 770. In some embodiments, the rules are stored as a setof flow tables that each includes a set of flow entries. The flowprocessor 775 handles packets for which integration bridge 750(described below) does not have a matching rule. In such cases, the flowprocessor 775 matches the packets against its stored rules. When apacket matches a rule, the flow processor 775 sends the matched rule andthe packet to the integration bridge 750 for the integration bridge 750to process. This way, when the integration bridge 750 receives a similarpacket that matches the generated rule, the packet will be matchedagainst the generated exact match rule in the integration bridge 750 andthe flow processor 775 will not have to process the packet.

In some embodiments, the flow processor 775 may not have a rule to whichthe packet matches. In such cases, some embodiments of the flowprocessor 775 send the packet to another managed switching element(e.g., a pool node) for handling packets that cannot be processed by anedge switching element. However, in other cases, the flow processor 775may have received from the network controller 780 a catchall rule thatdrops the packet when a rule to which the packet matches does not existin the flow processor 775.

As illustrated in FIG. 7, the kernel 720 includes a hypervisor networkstack 740 and an OVS kernel module 745. The hypervisor network stack 740is an Internet Protocol (IP) network stack in some embodiments. Thehypervisor network stack 740 processes and routes IP packets that arereceived from the OVS kernel module 745 and the PIF bridges 755 and 760.When processing a packet that is destined for a network host external tothe host 700, the hypervisor network stack 740 determines to which ofthe physical interface (PIF) bridges 755 and 760 the packet should besent.

The OVS kernel module 745 processes and routes network data (e.g.,packets) between VMs running on the host 700 and network hosts externalto the host 700 (e.g., network data received through the NICs 710 and715). In some embodiments, the OVS kernel module 745 implements theforwarding tables of the physical control plane for one or more logicalnetworks. To facilitate the processing and routing of network data, theOVS kernel module 745 communicates with OVS daemon 765 (e.g., to receiveflow entries from the OVS daemon 765). In some embodiments, the OVSkernel module 745 includes a bridge interface (not shown) that allowsthe hypervisor network stack 740 to send packets to and receive packetsfrom the OVS kernel module 745.

FIG. 7 illustrates that the OVS kernel module 745 includes anintegration bridge 750 and the PIF bridges 755 and 760. In someembodiments, the OVS kernel module 745 includes a PIF bridge for eachNIC in the hardware 705. In other embodiments, a PIF bridge in the OVSkernel module 745 may interact with more than one NIC in the hardware705. The PIF bridges 755 and 760 route network data between thehypervisor network stack 740 and network hosts external to the host 700(i.e., network data received through the NICs 710 and 715)

The integration bridge 750 processes and routes packets received fromthe hypervisor network stack 740, the VMs 735 and 738 (e.g., throughVIFs), and the PIF bridges 755 and 760. In some embodiments, theintegration bridge 750 stores a subset of the rules stored in the flowprocessor 775 (and/or rules derived from rules stored in the flowprocessor 775) that the integration bridge 750 is currently using or wasrecently using to process and forward packets.

In some embodiments, the flow processor 775 of some embodiments isresponsible for managing rules in the integration bridge 750. In someembodiments, the integration bridge 750 stores only active rules. Theflow processor 775 monitors the rules stored in the integration bridge750 and removes the active rules that have not been access for a definedamount of time (e.g., 1 second, 3 seconds, 5, seconds, 10 seconds,etc.). In this manner, the flow processor 775 manages the integrationbridge 750 so that the integration bridge 750 stores rules that arebeing used or have recently been used.

Although FIG. 7 illustrates one integration bridge, the OVS kernelmodule 745 may include multiple integration bridges. For instance, insome embodiments, the OVS kernel module 745 includes a separateintegration bridge for each logical switching element that isimplemented across a managed network to which the software switchingelement belongs. That is, the OVS kernel module 745 has a correspondingintegration bridge for each logical switching element that isimplemented across the managed network.

The above description relates to the forwarding functions of the managedsoftware switching element of some embodiments. Just as the softwareswitching element includes a user space component that implements thecontrol plane (the OVS daemon 765) and a kernel component thatimplements the data plane (the OVS kernel module 745), the distributedmiddlebox element of some embodiments includes a control plane componentoperating in the user space (the middlebox daemon 790) and a data planecomponent operating in the kernel (the middlebox kernel module 795).

As shown, the middlebox daemon 790 includes a middlebox configurationreceiver 791 and a middlebox configuration compiler 792. The middleboxconfiguration receiver 791 communicates with the network controller 780in order to receive the configuration for the middlebox, as well asslicing information. The middlebox configuration, in some embodiments,is a set of records (e.g., in the same form as flow entry recordsreceived by the OVS daemon) describing the middlebox packet processingrules. For example, a firewall configuration includes a set of packetprocessing rules describing when to drop packets, allow packets, etc.(similar to ACL entries, but also including TCP connection state as afactor in the decisions). A source network address translationconfiguration includes a set of hidden IP addresses of virtual machinesthat should be mapped into an exposed IP address by the translator. Aload balancer configuration, in some embodiments, includes the networkaddress translation mapping of an exposed IP address into severaldifferent hidden virtual machine addresses, as well as a load balancing(scheduling) algorithm for determining to which of several machines newTCP connections should be sent.

As described above, the slicing information assigns an identifier to aparticular middlebox instance to be performed by the distributedmiddlebox element. In some embodiments, the identifier is bound to aparticular logical middlebox in a particular tenant's logical network.That is, when a particular logical network includes several differentmiddleboxes with different processing rules, the middlebox daemon 790will create several middlebox instances. Each of these instances isidentified with a different slice identifier on packets sent to themiddlebox. In addition, in some embodiments the middlebox daemon 790assigns a particular internal identifier for each of these instances,which the middlebox uses in its internal processing (e.g., in order tokeep track of active TCP connections).

The middlebox daemon 790 also includes a middlebox configurationcompiler 792. In some embodiments, the middlebox configuration compiler792 receives the middlebox configuration (e.g., the packet processing,modification, or analysis rules) for a particular middlebox instance ina first language, and compiles these into a set of rules in a secondlanguage more optimized for the internal processing of the middlebox.The middlebox configuration compiler 792 sends the compiled packetprocessing rules to the middlebox processor 796 of the middlebox kernelmodule 795.

The middlebox kernel module 795 processes packets sent from and/or toVMs running on the host 700 in order to determine whether to allow thepackets through, drop the packets, etc. As shown, the middlebox kernelmodule 795 includes a middlebox processor 795 to perform thesefunctions. The middlebox processor 795 receives translated middleboxrules for a particular middlebox instance from the middleboxconfiguration compiler 792. In some embodiments, these translatedmiddlebox rules specify a packet processing pipeline within themiddlebox.

In order to receive packets from the managed switching element, themiddlebox processor 796 of some embodiments connects to a software portabstraction on the integration bridge 750 of the OVS kernel module.Through this port on the integration bridge, the managed switchingelement sends packets to the middlebox and receives packets from themiddlebox after processing by the middlebox (unless the middlebox dropsthe packet). As described, these packets include a slice identifier tagused by the middlebox processor 796 to determine which set of compiledpacket processing rules to apply to the packet.

The architectural diagram of the distributed middlebox and softwareswitching element illustrated in FIG. 7 is one exemplary configuration.One of ordinary skill in the art will recognize that otherconfigurations are possible. For instance, in some embodiments themiddlebox processor that applies the compiled packet processing rules islocated in the user space 721 rather than the kernel 720. In suchembodiments, the kernel exposes the network interfaces 710 and 715 forfull control by the user space so that the middlebox processor canperform its functions in the user space without a loss of speed ascompared to the kernel.

III. Network Control System

Section I above described the different middlebox implementationarchitectures, from fully distributed to fully centralized. Asmentioned, in some embodiments these middleboxes may be provisionedthrough a network control system that is also used to provision themanaged switching elements that implement the logical forwardingelements of the network. In some embodiments, the network control systemis a hierarchical set of network controllers.

FIG. 8 illustrates a network control system 800 of some embodiments forconfiguring managed switching elements and distributed middleboxelements in order to implement logical networks. As shown, the networkcontrol system 800 includes an input translation controller 805, alogical controller 810, physical controllers 815 and 820, hosts 825-840,and a centralized middlebox 845. As shown, the hosts 830-865 includeboth managed switching elements and middlebox elements, which may beimplemented as shown above in FIG. 7. One of ordinary skill in the artwill recognize that many other different combinations of the variouscontrollers and hosts are possible for the network control system 800.

In some embodiments, each of the controllers in a network control systemhas the capability to function as an input translation controller,logical controller, and/or physical controller. Alternatively, in someembodiments a given controller may only have the functionality tooperate as a particular one of the types of controller (e.g., as aphysical controller). In addition, different combinations of controllersmay run in the same physical machine. For instance, the inputtranslation controller 805 and the logical controller 810 may run in thesame computing device, with which a user interacts.

Furthermore, each of the controllers illustrated in FIG. 8 (andsubsequent FIG. 9) is shown as a single controller. However, each ofthese controllers may actually be a controller cluster that operates ina distributed fashion to perform the processing of a logical controller,physical controller, or input translation controller.

The input translation controller 805 of some embodiments includes aninput translation application that translates network configurationinformation received from a user. For example, a user may specify anetwork topology such as that shown in FIG. 2, which includes aspecification as to which machines belong in which logical domain. Thiseffectively specifies a logical data path set, or a set of logicalforwarding elements. For each of the logical switches, the userspecifies the machines that connect to the logical switch (i.e., towhich logical ports are assigned for the logical switch). In someembodiments, the user also specifies IP addresses for the machines. Theinput translation controller 805 translates the entered network topologyinto logical control plane data that describes the network topology. Forexample, an entry might state that a particular MAC address A is locatedat a particular logical port X of a particular logical switch.

In some embodiments, each logical network is governed by a particularlogical controller (e.g., logical controller 810). The logicalcontroller 810 of some embodiments translates the logical control planedata into logical forwarding plane data, and the logical forwardingplane data into universal control plane data. Logical forwarding planedata, in some embodiments, consists of flow entries described at alogical level. For the MAC address A at logical port X, logicalforwarding plane data might include a flow entry specifying that if thedestination of a packet matches MAC A, forward the packet to port X.

The universal physical control plane data of some embodiments is a dataplane that enables the control system of some embodiments to scale evenwhen it contains a large number of managed switching elements (e.g.,thousands) to implement a logical data path set. The universal physicalcontrol plane abstracts common characteristics of different managedswitching elements in order to express physical control plane datawithout considering differences in the managed switching elements and/orlocation specifics of the managed switching elements.

As stated, the logical controller 510 of some embodiments translateslogical control plane data into logical forwarding plane data (e.g.,logical flow entries), then translates the logical forwarding plane datainto universal control plane data. In some embodiments, the logicalcontroller application stack includes a control application forperforming the first translation and a virtualization application forperforming the second translation. Both of these applications, in someembodiments, use a rules engine for mapping a first set of tables into asecond set of tables. That is, the different data planes are representedas tables (e.g., n Log tables), and the controller applications use atable mapping engine to translate between the data planes.

Each of the physical controllers 815 and 820 is a master of one or moremanaged switching elements (e.g., located within host machines). In thisexample, each of the two physical controllers is a master of two managedswitching elements. Furthermore, the physical controller 815 is themaster of the centralized middlebox 845. In some embodiments, a physicalcontroller receives the universal physical control plane information fora logical network and translates this data into customized physicalcontrol plane information for the particular managed switches that thephysical controller manages. In other embodiments, the physicalcontroller passes the appropriate universal physical control plane datato the managed switch, which includes the ability (e.g., in the form ofa chassis controller running on the host machine) to perform theconversion itself.

The universal physical control plane to customized physical controlplane translation involves a customization of various data in the flowentries. For the example noted above, the universal physical controlplane would involve several flow entries. The first entry states that ifa packet matches the particular logical data path set (e.g., based onthe packet being received at a particular logical ingress port), and thedestination address matches MAC A, then forward the packet to logicalport X. This flow entry will be the same in the universal and customizedphysical control planes, in some embodiments. Additional flows aregenerated to match a physical ingress port (e.g., a virtual interface ofthe host machine) to the logical ingress port X (for packets receivedfrom MAC A, as well as to match logical port X to the particular egressport of the physical managed switch. However, these physical ingress andegress ports are specific to the host machine containing the managedswitching element. As such, the universal physical control plane entriesinclude abstract physical ports while the customized physical controlplane entries include the actual physical ports involved.

In some embodiments, the network control system also disseminates datarelating to the middleboxes of a logical network. The network controlsystem may disseminate middlebox configuration data, as well as datarelating to the sending and receiving of packets to/from the middleboxesat the managed switches and to/from the managed switches at themiddleboxes.

As shown in FIG. 8, the same network control system distributes data toboth distributed and centralized middleboxes in some embodiments.Several physical controllers are used to disseminate the configurationof a distributed middlebox, whereas some embodiments assign a specificphysical controller to a centralized middlebox appliance. In this case,the physical controller 815 is assigned to disseminate the configurationof the centralized middlebox 845, while the configuration for thedistributed middlebox is disseminated through both of the physicalcontrollers 815 and 820.

In order to incorporate the middleboxes, the flow entries propagatedthrough the network control system to the managed switches will includeentries for sending the appropriate packets to the appropriatemiddleboxes (e.g., flow entries that specify for packets having a sourceTP address in a particular subnet to be forwarded to a particularmiddlebox). In addition, the flow entries for the managed switch willneed to specify how to send such packets to the middleboxes. That is,once a first entry specifies a logical egress port of the logical routerto which a particular middlebox is bound, additional entries arerequired to attach the logical egress port to the middlebox.

For the centralized middlebox 845, these additional entries will matchthe logical egress port of the logical router to a particular physicalport of the host machine (e.g., a physical network interface) throughwhich the host machine connects to the middlebox. In addition, theentries include encapsulation information for sending the packet to thecentralized middlebox appliance via a tunnel between the host machineand the middlebox.

For the distributed middlebox, the packet does not have to actuallyleave the host machine in order to reach the middlebox. However, themanaged switching element nevertheless needs to include flow entries forsending the packet to the middlebox element on the host machine. Theseflow entries, again, include an entry to map the logical egress port ofthe logical router to the port through which the managed switchingelement connects to the middlebox. However, in this case the middleboxattaches to a software abstraction of a port in the managed switchingelement, rather than a physical (or virtual) interface of the hostmachine. That is, a port is created within the managed switchingelement, to which the middlebox element attaches. The flow entries inthe managed switching element send packets to this port in order for thepackets to be routed within the host machine to the middlebox.

For both the distributed and centralized middleboxes, in someembodiments the managed switching element adds slicing information tothe packet. Essentially, this slicing information is a tag thatindicates to which of the (potentially) several instances being run bythe middlebox the packet should be sent. Thus, when the middleboxreceives the packet, the tag enables the middlebox to use theappropriate set of packet processing, analysis, modification, etc. rulesin order to perform its operations on the packet. Some embodiments,rather than adding slicing information to the packet, either definedifferent ports of the managed switching element for each middleboxinstance, and essentially use the ports to slice the traffic destinedfor the firewall (in the distributed case), or connect to differentports of the centralized appliance to differentiate between theinstances (in the centralized case).

The above describes the propagation of the forwarding data to themanaged switching elements. In addition, some embodiments use thenetwork control system to propagate configuration data to themiddleboxes. FIG. 9 conceptually illustrates the propagation of datathrough the network control system of some embodiments. On the left sideof the figure is the data flow to the managed switching elements thatimplement a logical network, while the right side of the figure showsthe propagation of both middlebox configuration data as well as networkattachment and slicing data to the middleboxes.

On the left side, the input translation controller 805 receives anetwork configuration through an API, which is converted into logicalcontrol plane data. This network configuration data includes a logicaltopology such as that shown in FIG. 2. In addition, the networkconfiguration data of some embodiments includes routing policies thatspecify which packets are sent to the middlebox. When the middlebox islocated on a logical wire between two logical forwarding elements (e.g.,between a logical router and a logical switch), then all packets sentover that logical wire will automatically be forwarded to the middlebox.However, for an out-of-band middlebox such as that in networkarchitecture 200, the logical router will only send packets to themiddlebox when particular policies are specified by the user.

Whereas routers and switches will normally forward packets according tothe destination address (e.g., MAC address or IP address) of the packet,policy routing allows forwarding decisions to be made based on otherinformation stored by the packet (e.g., source addresses, a combinationof source and destination addresses, etc.). For example, the user mightspecify that all packets with source IP addresses in a particularsubnet, or that have destination IP addresses not matching a particularset of subnets, should be forwarded to the middlebox.

As shown, the logical control plane data is converted by the logicalcontroller 810 (specifically, by the control application of the logicalcontroller) to logical forwarding plane data, and then subsequently (bythe virtualization application of the logical controller) to universalphysical control plane data. In some embodiments, these conversionsgenerate a flow entry (at the logical forwarding plane), then adds amatch over the logical data path set (at the universal physical controlplane). The universal physical control plane also includes additionalflow entries for mapping generic physical ingress ports (i.e., a genericabstraction of a port not specific to any particular physical hostmachine) to logical ingress ports as well as for mapping logical egressports to generic physical egress ports. For instance, for the mapping toa centralized middlebox, the flow entries at the universal physicalcontrol plane would include a forwarding decision to send a packet tothe logical port to which the middlebox connects when a routing policyis matched, as well as a mapping of the logical port to a genericphysical port of a host machine that connects to the middlebox.

The physical controller 815 (one of the several physical controllers),as shown, translates the universal physical control plane data intocustomized physical control plane data for the particular managedswitching elements 830-840 that it manages. This conversion involvessubstituting specific data (e.g., specific physical ports) for thegeneric abstractions in the universal physical control plane data. Forinstance, in the example of the above paragraph, the port integrationentries are configured to specify the physical layer port appropriatefor the particular middlebox configuration. This port might be a virtualNIC if the firewall runs as a virtual machine on the host machine, orthe previously-described software port abstraction within the managedswitching element when the firewall runs as a process (e.g., daemon)within the hypervisor on the virtual machine. In some embodiments, forthe latter situation, the port is an IPC channel or TUN/TAP device-likeinterface. In some embodiments, the managed switching element includesone specific port abstraction for the firewall module and sends thisinformation to the physical controller in order for the physicalcontroller to customize the physical control plane flows. For the flowentries to send packets to the centralized middlebox 845, on the otherhand, the inserted port will be an actual physical port of theparticular host machine on which the managed switching element operates.

In addition, in some embodiments the physical controller adds flowentries specifying slicing information particular to the middlebox. Forinstance, for a particular managed switching element, the flow entry mayspecify to add a particular tag (e.g., a VLAN tag or similar tag) to apacket before sending the packet to the particular firewall. Thisslicing information enables the middlebox to receive the packet andidentify which of its several independent instances should process thepacket.

The managed switching element 825 (one of several MSEs managed by thephysical controller 815) performs a translation of the customizedphysical control plane data into physical forwarding plane data. Thephysical forwarding plane data, in some embodiments, are the flowentries stored within a switching element (either a physical router orswitch or a software switching element) against which the switchingelement actually matches received packets.

The right side of FIG. 9 illustrates two sets of data propagated to amiddleboxes (either the centralized or distributed middlebox) ratherthan the managed switching elements. The first of these sets of data isthe actual middlebox configuration data that includes various rulesspecifying the operation of the particular logical middlebox. This datamay be received at the input translation controller 805 or a differentinput interface, through an API particular to the middleboximplementation. In some embodiments, different middlebox implementationswill have different interfaces presented to the user (i.e., the userwill have to enter information in different formats for differentparticular middleboxes). As shown, the user enters a middleboxconfiguration, which is translated by the middlebox API into middleboxconfiguration data.

In some embodiments, the middlebox configuration data is a set ofrecords, with each record specifying a particular rule. These records,in some embodiments, are in a similar format to the flow entriespropagated to the managed switching elements. In fact, some embodimentsuse the same applications on the controllers to propagate the firewallconfiguration records as for the flow entries, and the same tablemapping language (e.g., n Log) for the records.

The middlebox configuration data, in some embodiments, is not translatedby the logical or physical controller, while in other embodiments thelogical and/or physical controller perform at least a minimaltranslation of the middlebox configuration data records. As manymiddlebox packet processing, modification, and analysis rules operate onthe IP address (or TCP connection state) of the packets, and the packetssent to the middlebox will have this information exposed (i.e., notencapsulated within the logical port information), the middleboxconfiguration does not require translation from logical to physical dataplanes. Thus, the same middlebox configuration data is passed from theinput translation controller 805 (or other interface), to the logicalcontroller 810, to the physical controller 815.

In some embodiments, the logical controller 810 stores a description ofthe logical network and of the physical implementation of that physicalnetwork. The logical controller receives the one or more middleboxconfiguration records for a distributed middlebox, and identifies whichof the various nodes (i.e., host machines) will need to receive theconfiguration information. In some embodiments, the entire middleboxconfiguration is distributed to middlebox elements at all of the hostmachines, so the logical controller identifies all of the machines onwhich at least one virtual machine resides whose packets require use ofthe middlebox. This may be all of the virtual machines in a network(e.g., as for the middlebox shown in FIG. 2), or a subset of the virtualmachines in the network (e.g., when a firewall is only applied totraffic of a particular domain within the network). Some embodimentsmake decisions about which host machines to send the configuration datato on a per-record basis. That is, each particular rule may apply onlyto a subset of the virtual machines, and only hosts running thesevirtual machines need to receive the record.

Once the logical controller identifies the particular nodes to receivethe records, the logical controller identifies the particular physicalcontrollers that manage these particular nodes. As mentioned, each hostmachine has an assigned master physical controller. Thus, if the logicalcontroller identifies only first and second hosts as destinations forthe configuration data, the physical controllers for these hosts will beidentified to receive the data from the logical controller (and otherphysical controllers will not receive this data). For a centralizedmiddlebox, the logical controller needs only to identify the (single)physical controller that manages the appliance implementing themiddlebox.

In order to supply the middlebox configuration data to the hosts, thelogical controller of some embodiments pushes the data (using an exportmodule that accesses the output of the table mapping engine in thelogical controller) to the physical controllers. In other embodiments,the physical controllers request configuration data (e.g., in responseto a signal that the configuration data is available) from the exportmodule of the logical controller.

The physical controllers pass the data to the middlebox elements on thehost machines that they manage, much as they pass the physical controlplane data. In some embodiments, the middlebox configuration and thephysical control plane data are sent to the same database running on thehost machine, and the managed switching element and middlebox moduleretrieve the appropriate information from the database. Similarly, for acentralized middlebox 845, the physical controller 815 passes themiddlebox configuration data to the middlebox appliance (e.g., to adatabase for storing configuration data).

In some embodiments, the middlebox translates the configuration data.The middlebox configuration data will be received in a particularlanguage to express the packet processing, analysis, modification, etc.rules. The middlebox (distributed and/or centralized) of someembodiments compiles these rules into more optimized packetclassification rules. In some embodiments, this transformation issimilar to the physical control plane to physical forwarding plane datatranslation. When a packet is received by the middlebox, it applies thecompiled optimized rules in order to efficiently and quickly perform itsoperations on the packet.

In addition to the middlebox configuration rules, the middlebox modulesreceive slicing and/or attachment information in order to receivepackets from and send packets to the managed switching elements. Thisinformation corresponds to the information sent to the managed switchingelements. As shown, in some embodiments the physical controller 815generates the slicing and/or attachment information for the middlebox(i.e., this information is not generated at the input or logicalcontroller level of the network control system).

For distributed middleboxes, the physical controllers, in someembodiments, receive information about the software port of the managedswitching element to which the middlebox connects from the managedswitching element itself, then passes this information down to themiddlebox. In other embodiments, however, the use of this port iscontracted directly between the middlebox module and the managedswitching element within the host machine, so that the middlebox doesnot need to receive the attachment information from the physicalcontroller. In some such embodiments, the managed switching elementnevertheless transmits this information to the physical controller inorder for the physical controller to customize the universal physicalcontrol plane flow entries for receiving packets from and sendingpackets to the middlebox.

For centralized middleboxes, some embodiments provide tunnelingattachment data to the middlebox. The middlebox, in some embodiments,will need to know the type of tunnel encapsulation various host machineswill use to send packets to the middlebox. In some embodiments, themiddlebox has a list of accepted tunneling protocols (e.g., STT, GRE,etc.), and the chosen protocol is coordinated between the managedswitching element(s) and the middlebox. The tunneling protocol may beentered by the user as part of the middlebox configuration, or may beautomatically determined by the network control system in differentembodiments. In addition to the connections to the host machines, atunnel will be set up between the centralized middlebox and the poolnode to which it sends packets after processing, as described byreference to FIG. 4 above.

The slicing information generated by the physical controller, in someembodiments, consists of an identifier for the middlebox instance to beused for the particular logical network. In some embodiments, asdescribed, the middlebox, whether operating on the host machine or as acentralized appliance, is virtualized for use by multiple logicalnetworks. When the middlebox receives a packet from the managedswitching element, in some embodiments the packet includes a prependedtag (e.g., similar to a VLAN tag) that identifies a particular one ofthe middlebox instances (i.e., a particular configured set of rules) touse in processing the packet.

As shown in FIG. 9, the middlebox translates this slicing informationinto an internal slice binding. In some embodiments, the middlebox usesits own internal identifiers (different from the tags prepended to thepackets) in order to identify states (e.g., active TCP connections,statistics about various IP addresses, etc.) within the middlebox. Uponreceiving an instruction to create a new middlebox instance and anexternal identifier (that used on the packets) for the new instance,some embodiments automatically create the new middlebox instance andassign the instance an internal identifier. In addition, the middleboxstores a binding for the instance that maps the external sliceidentifier to the internal slice identifier.

The above figures illustrate various physical and logical networkcontrollers. FIG. 10 illustrates example architecture of a networkcontroller (e.g., a logical controller or a physical controller) 1000.The network controller of some embodiments uses a table mapping engineto map data from an input set of tables to data in an output set oftables. The input set of tables in a controller include logical controlplane (LCP) data to be mapped to logical forwarding plane (LFP) data,LFP data to be mapped to universal physical control plane (UPCP) data,and/or UPCP data to be mapped to customized physical control plane(CPCP) data. The input set of tables may also include middleboxconfiguration data to be sent to another controller and/or a distributedmiddlebox instance. The network controller 1000, as shown, includesinput tables 1015, a rules engine 1010, output tables 1020, an importer1030, an exporter 1035, a translator 1035, and a persistent data storage(PTD) 1040.

In some embodiments, the input tables 1015 include tables with differenttypes of data depending on the role of the controller 1000 in thenetwork control system. For instance, when the controller 1000 functionsas a logical controller for a user's logical forwarding elements, theinput tables 1015 include LCP data and LFP data for the logicalforwarding elements. When the controller 1000 functions as a physicalcontroller, the input tables 1015 include LFP data. The input tables1015 also include middlebox configuration data received from the user oranother controller. The middlebox configuration data is associated witha logical datapath set parameter that identifies the logical switchingelements to which the middlebox to be is integrated.

In addition to the input tables 1015, the control application 1000includes other miscellaneous tables (not shown) that the rules engine1010 uses to gather inputs for its table mapping operations. Thesemiscellaneous tables include constant tables that store defined valuesfor constants that the rules engine 1010 needs to perform its tablemapping operations (e.g., the value 0, a dispatch port number forresubmits, etc.). The miscellaneous tables further include functiontables that store functions that the rules engine 1010 uses to calculatevalues to populate the output tables 1025.

The rules engine 1010 performs table mapping operations that specifiesone manner for converting input data to output data. Whenever one of theinput tables is modified (referred to as an input table event), therules engine performs a set of table mapping operations that may resultin the modification of one or more data tuples in one or more outputtables.

In some embodiments, the rules engine 1010 includes an event processor(not shown), several query plans (not shown), and a table processor (notshown). Each query plan is a set of rules that specifies a set of joinoperations that are to be performed upon the occurrence of an inputtable event. The event processor of the rules engine 1010 detects theoccurrence of each such event. In some embodiments, the event processorregisters for callbacks with the input tables for notification ofchanges to the records in the input tables 1015, and detects an inputtable event by receiving a notification from an input table when one ofits records has changed.

In response to a detected input table event, the event processor (1)selects an appropriate query plan for the detected table event, and (2)directs the table processor to execute the query plan. To execute thequery plan, the table processor, in some embodiments, performs the joinoperations specified by the query plan to produce one or more recordsthat represent one or more sets of data values from one or more inputand miscellaneous tables. The table processor of some embodiments then(1) performs a select operation to select a subset of the data valuesfrom the record(s) produced by the join operations, and (2) writes theselected subset of data values in one or more output tables 1020.

Some embodiments use a variation of the datalog database language toallow application developers to create the rules engine for thecontroller, and thereby to specify the manner by which the controllermaps logical datapath sets to the controlled physical switchinginfrastructure. This variation of the datalog database language isreferred to herein as n Log. Like datalog, n Log provides a fewdeclaratory rules and operators that allow a developer to specifydifferent operations that are to be performed upon the occurrence ofdifferent events. In some embodiments, n Log provides a limited subsetof the operators that are provided by datalog in order to increase theoperational speed of n Log. For instance, in some embodiments, n Logonly allows the AND operator to be used in any of the declaratory rules.

The declaratory rules and operations that are specified through n Logare then compiled into a much larger set of rules by an n Log compiler.In some embodiments, this compiler translates each rule that is meant toaddress an event into several sets of database join operations.Collectively the larger set of rules forms the table mapping rulesengine that is referred to as the n Log engine.

Some embodiments designate the first join operation that is performed bythe rules engine for an input event to be based on the logical datapathset parameter. This designation ensures that the rules engine's joinoperations fail and terminate immediately when the rules engine hasstarted a set of join operations that relate to a logical datapath set(i.e., to a logical network) that is not managed by the controller.

Like the input tables 1015, the output tables 1020 include tables withdifferent types of data depending on the role of the controller 1000.When the controller 1000 functions as a logical controller, the outputtables 1015 include LFP data and UPCP data for the logical switchingelements. When the controller 1000 functions as a physical controller,the output tables 1020 include CPCP data. Like the input tables, theoutput tables 1015 may also include the middlebox configuration data.Furthermore, the output tables 1015 may include a slice identifier whenthe controller 1000 functions as a physical controller.

In some embodiments, the output tables 1020 can be grouped into severaldifferent categories. For instance, in some embodiments, the outputtables 1020 can be rules engine (RE) input tables and/or RE outputtables. An output table is a RE input table when a change in the outputtable causes the rules engine to detect an input event that requires theexecution of a query plan. An output table can also be an RE input tablethat generates an event that causes the rules engine to perform anotherquery plan. An output table is a RE output table when a change in theoutput table causes the exporter 1025 to export the change to anothercontroller or a MSE. An output table can be an RE input table, a REoutput table, or both an RE input table and a RE output table.

The exporter 1025 detects changes to the RE output tables of the outputtables 1020. In some embodiments, the exporter registers for callbackswith the RE output tables for notification of changes to the records ofthe RE output tables. In such embodiments, the exporter 1025 detects anoutput table event when it receives notification from a RE output tablethat one of its records has changed.

In response to a detected output table event, the exporter 1025 takeseach modified data tuple in the modified RE output tables and propagatesthis modified data tuple to one or more other controllers or to one ormore MSEs. When sending the output table records to another controller,the exporter in some embodiments uses a single channel of communication(e.g., a RPC channel) to send the data contained in the records. Whensending the RE output table records to MSEs, the exporter in someembodiments uses two channels. One channel is established using a switchcontrol protocol (e.g., OpenFlow) for writing flow entries in thecontrol plane of the MSE. The other channel is established using adatabase communication protocol (e.g., JSON) to send configuration data(e.g., port configuration, tunnel information).

In some embodiments, the controller 1000 does not keep in the outputtables 1020 the data for logical datapath sets that the controller isnot responsible for managing (i.e., for logical networks managed byother logical controllers). However, such data is translated by thetranslator 1035 into a format that can be stored in the PTD 1040 and isthen stored in the PTD. The PTD 1040 propagates this data to PTDs of oneor more other controllers so that those other controllers that areresponsible for managing the logical datapath sets can process the data.

In some embodiments, the controller also brings the data stored in theoutput tables 1020 to the PTD for resiliency of the data. Therefore, inthese embodiments, a PTD of a controller has all the configuration datafor all logical datapath sets managed by the network control system.That is, each PTD contains the global view of the configuration of thelogical networks of all users.

The importer 1030 interfaces with a number of different sources of inputdata and uses the input data to modify or create the input tables 1010.The importer 1020 of some embodiments receives the input data fromanother controller. The importer 1020 also interfaces with the PTD 1040so that data received through the PTD from other controller instancescan be translated and used as input data to modify or create the inputtables 1010. Moreover, the importer 1020 also detects changes with theRE input tables in the output tables 1030.

IV. Example Implementation of Several Middleboxes

The above describes various principles for implementing and configuringboth distributed and centralized middleboxes. The example network shownin FIG. 2 above is a simplified example with only a single middlebox.FIG. 11, on the other hand, conceptually illustrates a more complexlogical network topology 1100 involving numerous middleboxes.

The logical network 1100 includes three logical L2 domains: web servers1105-1115 connected to a first logical L2 switch 1140, applicationservers 1120 and 1125 connected to a second logical L2 switch 1145, anddata servers 1130 and 1135 connected to a third logical switch 1150.Each of these logical switches 1140-1150 connects to a logical router1155 (through various middleboxes).

Between each logical switch and the logical router 1155 is a loadbalancer, in order for the load balancer to schedule incoming traffic tothe particular logical L2 domain. That is, the first load balancer 1160performs destination network address translation (D-NAT) to balancetraffic (e.g., on a per transport connection basis) between the threeweb servers 1105-1115, the second load balancer 1165 performs D-NAT tobalance traffic between the two application servers 1120 and 1125, andthe third load balancer 1170 performs D-NAT to balance traffic betweenthe two data servers 1130 and 1135. In addition, between the logicalrouter 1155 and the second logical switch 1145 is a firewall 1175.

The logical router 1155 also connects the three logical L2 domains to anexternal network 1195, from which client requests may come into thenetwork. In addition, three middleboxes hang off of the L3 router 1155for processing traffic between the managed network and the externalnetwork. These middleboxes include a firewall 1180 for processingincoming traffic, and a source NAT 1185 for converting the real IPaddresses of outgoing traffic into one or more virtual IP addresses.These middleboxes are effectively located between the managed networkand the external network; however, because the physical implementationinvolves sending a packet to the middlebox and then receiving a packetback from the middlebox (either to send to the external network 1195 orthe appropriate host machine), the logical topology illustrates thesemiddleboxes as out-of-band middleboxes hanging off the router. Finally,an IDS 1190 also hangs off of the logical router 1155. In the network1100, the logical router forwards a duplicate of all processed packetsto the IDS 1190 for analysis.

FIG. 12 conceptually illustrates one particular physical implementation1200 of the network 1100 in a hosted, virtualized environment. As shown,the seven virtual machines 1105-1135 are distributed across fivedifferent host machines 1205-1225. Some of the host machines only hostone virtual machine, while other host two VMs. The host machines1205-1225 connect to each other as well as a pool node 1230, whichconnects to a gateway (also referred to as an extender) 1235. Thegateway 1235 connects the managed network 1200 to an external network1240 (e.g., the Internet, a different managed network, an externalprivate network, etc.). While this example illustrates the gateway 1230as only connected to the host machines 1205-1225 through the pool node1230, some embodiments implement a direct connection between the gatewayand the host machines.

As shown, each host machine 1205-1225, as well as the pool node 1230 andthe gateway 1235, includes a managed switching element. All of the hostmachines 1205-1225 are configured to include the flow entries for thelogical router 1155 as well as the logical switches 1140-1150. Thus, thesecond host machine 1210 that includes an application server 1125 aswell as a web server 1110 and the fifth host machine 1225 that onlyincludes a data server 1130 implement the same managed switchingelements. The managed switching element in the gateway 1235 implementsthe logical router 1155 as well as all three of the logical switches1140-1150 of the logical network 1100. The pool node, in someembodiments, is also a managed switching element that implements the L3router 1155 and all three logical switches 1140-1150.

In the implementation 1200, some of the logical middleboxes aredistributed, while others are centralized. For instance, the intrusiondetection service 1190 is implemented as a centralized IDS appliance1245. Each of the host machines 1205-1225 connects directly to the IDSappliance 1245, as does the gateway 1235. As shown, these machines onlysend packets to the IDS appliance, and do not receive packets back. Thisis because the IDS only receives duplicate packets (incoming packetsfrom the gateway 1235, and outgoing packets from the host machines1205-1225), and performs analysis to detect threats, but does not sendthe packets anywhere after analyzing them.

The S-NAT middlebox 1185 is distributed among each of the host machines(e.g., as a daemon running within the hypervisor), as all of the virtualmachines may send packets to the external network that require an IPaddress translation to hide the real IPs behind a virtual IP address.The firewall 1175 is distributed as well, but only implemented on thehost machines 1210 and 1220, because these are the nodes that host theapplication server virtual machines 1120 and 1125 that are behind thisfirewall.

The three load balancers 1160-1170 are implemented across the varioushost machines 1205-1225, as well as the gateway 1235. As shown, the loadbalancers are implemented within a load balancer element in each of thehost machines, such that the load balancer element is virtualized (i.e.,sliced) to implement several different load balancers. Whereas thefirewall 1175 is located at the host machines where the applicationservers are each implemented, each particular logical load balancer islocated on each node that hosts a machine for which the particular loadbalancer is not responsible. This is because, for example, the loadbalancer 1170 receives any packet destined for the virtual IP addressrepresenting the data servers 1130 and 1135, determines to which of thetwo data servers to forward the packet, then modifies the destination IPaddress to reflect the selected data server. As the processing isperformed at the first hop (packet source) whenever possible, thisfunctionality is not needed at the nodes hosting the data servers(unless other virtual machines are also hosted), but rather at the nodeshosting the other virtual machines that may send packets to the dataservers 1130 and 1135. Accordingly, the load balancer element in thegateway 1235 is sliced to implement all three load balancers, asincoming packets from the external network 1240 may be destined for anyof the three logical L2 domains.

In addition, as shown, some embodiments implement any distributedmiddleboxes for the logical network within the pool node 1230 and thegateway 1235 as well. Because it may be difficult to determine at theoutset which middleboxes will be needed at which locations (and a usermay subsequently modify routing policies, middlebox configuration, ornetwork architecture), some embodiments do not presume that certainphysical machines will not require a particular middlebox. Along thesame lines, some embodiments do not distribute different middleboxes todifferent subsets of the hosts. Instead, all middleboxes for the logicalnetwork are implemented at each host at which the logical network ispresent.

The firewall 1180, for processing incoming packets between the externalnetwork 1180 and the managed network, is implemented in a centralizedfashion in a virtual machine (rather than a module running in ahypervisor) in the gateway. When the gateway receives incoming packets,it automatically routes the packets to the firewall VM in someembodiments. While in this example the firewall VM is located in thegateway, some embodiments implement the firewall as a virtual machine ina host machine (e.g., a different host machine than those hosting theuser VMs), or using a firewall appliance.

In order to configure the network 1100, in some embodiments a userenters the network topology into a logical controller (e.g., via aninput translation controller) as shown above by reference to FIGS. 8 and9. In some embodiments, the user enters the connections between theswitches, routers, middleboxes, and virtual machines. Based on thelocations of the various network components, the input translationcontroller or logical controller generates logical control plane dataand converts this into flow entries in the logical forwarding plane.However, for middleboxes such as the firewall 1180, S-NAT 1185, and IDS1190, the user also has to enter policy routing rules indicating when tosend packets to these components. For instance, the routing policy forthe firewall 1180 would be to send all packets with a source IP outsideof the logical network, while the routing policy for the S-NAT 1185would be to send all packets with a source IP in the logical network.The logical controller, after converting the flow entries to theuniversal physical control plane, identifies which physical controllersshould receive which flow entries, and then distributes these flowentries. The physical controllers add the specific port information(unless the host machines include chassis controllers that perform theuniversal to customized physical control plane translation) and othercustomizations to the flow entries and distribute them to the managedswitching elements.

In addition, the user enters the middlebox configurations for thevarious load balancers, firewalls, etc. For example, this informationwould include the scheduling algorithms to use for each of the differentload balancers, the virtual IP to real IP mappings for the S-NAT, packetprocessing rules for the firewalls, etc. The users enter thisinformation through APIs for the various middleboxes, and someembodiments convert this information to records having the same format(e.g., n Log) as the flow entries. The logical controllers identifywhich middlebox configurations need to be sent to which host machines orcentralized middlebox appliances (e.g., the records for the firewall1175 only need to go to the host machines 1210 and 1220, while the S-NATrecords go to all five host machines 1205-1225). The logical controllerdistributes the records to the appropriate physical controllers, whichadd slicing information (and, in some cases, tunneling information) anddistribute the information to the middleboxes as described above.

The operation of the network will be described by reference to packetsincoming from the external network, packets outgoing to the externalnetwork, and packets sent from one logical L2 domain to the other. Whena packet is sent from a host machine (e.g., a web server), it firstarrives at the MSE running on the host. The packet will first enterlogical L2 processing by the logical switch, which sends the packet tothe logical router (because the packet is outgoing it does not need tobe sent to the local load balancer). The logical router (also handled bythe MSE at the host machine) sends a duplicate of the packet to the IDSappliance 1245, in addition to sending the packet to the S-NATprocessing on the host. The S-NAT processing modifies the source IPaddress and returns a new packet to the MSE. In some embodiments, if thepacket is part of an active TCP session, the S-NAT may have already sentflow entries to the MSE enabling the MSE to perform the modificationwithout the S-NAT processing being involved. The logical routerimplemented by the MSE then identifies the logical egress port as theport facing the external network, which maps to the physical port tosend the packet to the pool node 1230. The pool node forwards the packetto the gateway 1235, which sends the packet out to the external network1240.

When a packet is received at the gateway 1240 from the external network1240, the switching element processing in the gateway first sends thepacket to the firewall virtual machine. If the firewall does not dropthe packet, then the packet is returned to the switching elementprocessing, which identifies the correct slice of the load balancer,tags the packet with this slice information, and sends the packet to theload balancer processing. The load balancer selects a real destinationIP and sends a new packet with the IP address modified to reflect theselected destination machine. At this point, the MSE in the gatewaysends the packet to the correct host. If the destination VM is one ofthe application servers 1120 or 1125, the logical router in the MSE ofthe gateway first sends the packet to the firewall 1175 for processing,and then sends the packet to the correct host after receiving it backfrom the firewall element. The MSE then delivers the packet to thedestination machine.

Packets traveling from one logical domain to the other will not need totravel through the gateway 1235. The packet is initially received at theMSE, which performs L2 switching and then the L3 routing. The logicalrouter identifies the destination domain and tags the packet with thecorrect load balancer slicing information, then sends the tagged packetto the load balancer. The load balancer modifies the destination IP andreturns the packet to the MSE, which then routes the packet to thecorrect host machine for delivery to the VM.

V. Electronic System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or morecomputational or processing unit(s) (e.g., one or more processors, coresof processors, or other processing units), they cause the processingunit(s) to perform the actions indicated in the instructions. Examplesof computer readable media include, but are not limited to, CD-ROMs,flash drives, random access memory (RAM) chips, hard drives, erasableprogrammable read-only memories (EPROMs), electrically erasableprogrammable read-only memories (EEPROMs), etc. The computer readablemedia does not include carrier waves and electronic signals passingwirelessly or over wired connections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storagewhich can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 13 conceptually illustrates an electronic system 1300 with whichsome embodiments of the invention are implemented. The electronic system1300 may be a computer (e.g., a desktop computer, personal computer,tablet computer, etc.), server, dedicated switch, phone, PDA, or anyother sort of electronic or computing device. Such an electronic systemincludes various types of computer readable media and interfaces forvarious other types of computer readable media. Electronic system 1300includes a bus 1305, processing unit(s) 1310, a system memory 1325, aread-only memory 1330, a permanent storage device 1335, input devices1340, and output devices 1345.

The bus 1305 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 1300. For instance, the bus 1305 communicativelyconnects the processing unit(s) 1310 with the read-only memory 1330, thesystem memory 1325, and the permanent storage device 1335.

From these various memory units, the processing unit(s) 1310 retrievesinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 1330 stores static data and instructions thatare needed by the processing unit(s) 1310 and other modules of theelectronic system. The permanent storage device 1335, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system1300 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 1335.

Other embodiments use a removable storage device (such as a floppy disk,flash memory device, etc., and its corresponding drive) as the permanentstorage device. Like the permanent storage device 1335, the systemmemory 1325 is a read-and-write memory device. However, unlike storagedevice 1335, the system memory 1325 is a volatile read-and-write memory,such a random access memory. The system memory 1325 stores some of theinstructions and data that the processor needs at runtime. In someembodiments, the invention's processes are stored in the system memory1325, the permanent storage device 1335, and/or the read-only memory1330. From these various memory units, the processing unit(s) 1310retrieves instructions to execute and data to process in order toexecute the processes of some embodiments.

The bus 1305 also connects to the input and output devices 1340 and1345. The input devices 1340 enable the user to communicate informationand select commands to the electronic system. The input devices 1340include alphanumeric keyboards and pointing devices (also called “cursorcontrol devices”), cameras (e.g., webcams), microphones or similardevices for receiving voice commands, etc. The output devices 1345display images generated by the electronic system or otherwise outputdata. The output devices 1345 include printers and display devices, suchas cathode ray tubes (CRT) or liquid crystal displays (LCD), as well asspeakers or similar audio output devices. Some embodiments includedevices such as a touchscreen that function as both input and outputdevices.

Finally, as shown in FIG. 13, bus 1305 also couples electronic system1300 to a network 1365 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 1300 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself. In addition, someembodiments execute software stored in programmable logic devices(PLDs), ROM, or RAM devices.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium,” “computer readable media,” and “machinereadable medium” are entirely restricted to tangible, physical objectsthat store information in a form that is readable by a computer. Theseterms exclude any wireless signals, wired download signals, and anyother ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. Thus, one of ordinary skill in the artwould understand that the invention is not to be limited by theforegoing illustrative details, but rather is to be defined by theappended claims.

We claim:
 1. A system for implementing a logical network tocommunicatively connect a plurality of end machines, the logical networkcomprising (i) a set of logical forwarding elements collectivelyimplemented by a set of managed forwarding elements and (ii) at leasttwo logical middleboxes, the system comprising: a plurality of hostcomputers on each of which (i) a managed forwarding element executes toimplement the set of logical forwarding elements and (ii) a middleboxelement executes to implement a first logical middlebox of the logicalnetwork, wherein the middlebox elements collectively implement the firstlogical middlebox and each store state information for the first logicalmiddlebox but do not communicate the state information with the othermiddlebox elements; and a set of separate physical middleboxes forimplementing a second logical middlebox of the logical network, whereinthe second logical middlebox performs an operation that requires stateinformation relating to packets between several different sets of endmachines connected by the logical network and the set of separatephysical middleboxes share the state information for the second logicalmiddlebox with each other; wherein the middlebox elements and the set ofseparate physical middleboxes perform middlebox services on packetsbetween the end machines of the logical network.
 2. The system of claim1, wherein each particular middlebox element stores state informationregarding transport connections the packets of which are processed bythe particular middlebox element.
 3. The system of claim 2, wherein theprocessing for a particular transport connection does not require anystate information regarding other transport connections.
 4. The systemof claim 1, wherein the set of separate physical middleboxes comprises acluster of middleboxes that operate as a resource pool.
 5. The system ofclaim 4 further comprising a dedicated network connection between thecluster of middleboxes for sharing the state information between thephysical middleboxes in the cluster.
 6. The system of claim 1, whereinthe first logical middlebox comprises one of a firewall, a loadbalancer, and a source network address translator.
 7. The system ofclaim 1, wherein the second logical middlebox comprises an intrusiondetection system that performs operations requiring state informationfor each packet processed.
 8. The system of claim 1, wherein the secondlogical middlebox comprises a wide area network optimizer.
 9. A methodfor implementing a logical network to communicatively connect aplurality of end machines, the method comprising: receiving a logicalnetwork configuration, the logical network comprising (i) a set oflogical forwarding elements collectively implemented by a set of managedforwarding elements, (ii) a first logical middlebox, and (iii) a secondlogical middlebox; distributing configuration data to configuremiddlebox elements, executing on a plurality of host computers, tocollectively implement the first logical middlebox, wherein each of themiddlebox elements stores state information for the first logicalmiddlebox but does not communicate the state information with the othermiddlebox elements; and distributing configuration data to configure aset of separate physical middleboxes to implement a second logicalmiddlebox comprising operations that require state information relatingto packets between several different sets of end machines connected bythe logical network, wherein the set of separate physical middleboxesshare state information for the second logical middlebox with eachother, wherein the configured middlebox elements and the set of separatephysical middleboxes perform middlebox services on packets between theend machines of the logical network.
 10. The method of claim 9, whereineach particular middlebox element stores state information regardingtransport connections the packets of which are processed by theparticular middlebox element.
 11. The method of claim 10, wherein theprocessing by the first logical middlebox for a particular transportconnection does not require state information regarding other transportconnections.
 12. The method of claim 9, wherein the set of separatephysical middleboxes comprises a cluster of middleboxes that operate asa resource pool to implement the second logical middlebox.
 13. Themethod of claim 12, wherein a dedicated network connection existsbetween the physical middleboxes of the cluster for sharing the stateinformation regarding the second logical middlebox.
 14. The method ofclaim 9, wherein the first logical middlebox comprises one of afirewall, a load balancer, and a network address translator.
 15. Themethod of claim 9, wherein the second logical middlebox comprises anintrusion detection system that performs operations requiring stateinformation for each packet processed.
 16. The method of claim 9,wherein the second logical middlebox comprises a wide area networkoptimizer.
 17. The method of claim 9, wherein the logical network is afirst logical network, wherein the set of separate physical middleboxesalso implements a third logical middlebox that belongs to a secondlogical network.
 18. The method of claim 9, wherein the logical networkis a first logical network, wherein a subset of the middlebox elementsalso collectively implement a third logical middlebox that belongs to asecond logical network.